Deliberative Polls: toward Improved Measures of "informed" Public Opinion?

New research techniques have recently been developed to gather measures of public opinion that is better informed or more deliberative than that recorded in typical mass opinion surveys. These techniques include deliberative polls, educational surveys, and citizen planning cells. In view of what they set out to accomplish, what can we say from a scientific perspective about the utility of these methods? How are we to best interpret the data they produce? To address these questions, this paper reviews several of the most prominent and well-developed examples of deliberative or educational polling. We argue two main points. First, these new methods of assessing public opinion must be evaluated in terms of specific quality criteria that apply to different phases and/or participants in the democratic decision making process. Some techniques attempt to maximize several distinct qualities at once, making it difficult to identify specific objectives for evaluating success. Second, at least five important core methodological elements of educational or deliberative polls can be identified, each of which can theoretically alter results. To date, however, data bearing upon the effects of these methodological elements are in limited supply. Lack of knowledge about how method influences individual and collective opinion outcomes thus renders several of these techniques problematic. Survey researchers and analysts of public opinion have long debated the conceptual relationship between public opinion and mass survey data. Althou-h public opinion was classically conceived as an emergent product of widespr ..d popular deliberation and discussion (e.g. Bryce 1888, Blumer 1946, Habern.as 1989), many have come over time to accept mass survey data as its standard rendering (Key 1961, Converse 1987). However, in the face of evidence that opinions given to pollsters and survey researchers are often unorganized, isolated, uninformed responses (e.g. Bishop el al. 1980, Neuman 1986), arguments over * Portions of this manuscript were presented at the World Association for Public Opinion Research thematic seminar on Quality Criteria in Survey Research, Cadenabbia, Italy, June 1996. Work on this project was supported in part by the Marsh Center for the Study of Journalistic Performance in the Department of Communication Studies at the University of Michigan. © World Association for Public Opinion Research igg8 146 INTERNATIONAL JOURNAL OF PUBLIC OPINION RESEARCH the adequacy of mass survey data as indicators of deliberative public opinion persisted. In recent years 'deliberative polls' and variants of focused group discussions have been advanced as supplements or alternatives to conventional mass opinion surveys. These new efforts are intended to correct perceived problems of superficiality in mass opinion data and deficiencies in public communication. Some researchers have attempted to assemble scientifically representative samples of citizens who are immersed in an extensive program of deliberation and discussion, in order to produce measures of better-informed public opinion grounded in meaningful public discourse (e.g. Fishkin 1991). Other researchers have proposed less drastic modifications of standard survey research techniques in 'educational surveys' that attempt to inform respondents about complex public issues in an effort to gather assessments of more thoughtful opinion (e.g. Neijens 1987). How can the success of these efforts to gather measures of higher quality public opinion be judged? We attempt here to address the following questions. First, to what ends do these new methods aspire? What quality objectives, in particular, do they have in mind? Second, in view of what they set out to accomplish, what can we say from a scientific perspective about the utility of these methods? How well have they been evaluated in light of their quality objectives? How are we to best interpret the data they produce? What methodological conditions must be met to permit sensible interpretation of results? Since these techniques all, in one way or another, involve manipulation of respondents, what strictures must accompany interpretation of the resulting data? NEW METHODS FOR ASSESSING PUBLIC OPINION In a companion piece (Price and Neijens 1997), we have argued that conceptions of quality in public opinion are inextricably tied to broader conceptions of quality in democratic decision making, a complex process involving multiple phases and collective participants. Judgments of quality in public decision making can be conceptualized along a variety of dimensions. Decision theorists, who systematically examine and seek to improve human decision making, have suggested at least two dimensions relevant to judging the quality of any decision (e.g. Rohrmann 1986). One can judge the quality of the decision making process employed, or the quality of the decision outcome that is eventually attained. In the case of collective decision making, one can also apply quality considerations at both the individual level (e.g. How well do people understand the problems or proposed solutions at hand? Do they evaluate them rationally? Do they hold views that are in tfteir best interests?); and at the collective level (e.g. How completely has a society examined its options? DELIBERATIVE POLLS 147 How rationally is the collective choice determined? Is the choice in the best collective interest?). We can thus evaluate the quality of public opinion along each of these dimensions, which we have attempted to catalogue (a descriptive review of these criteria is provided by Price and Neijens 1997). Outcome-oriented criteria focus on the quality of opinions and decisions themselves. Many of these—for example, opinion stability, consistency, conviction, and knowledge of and acceptance of a decision's consequences—can be applied to both individual opinions and to aggregate public opinion. Other criteria—for instance the representativeness of an opinion—make sense mainly in connection with collective opinion. Process-oriented criteria can similarly be applied at both the collective and individual level. These include the extensiveness of the information search supporting an opinion, the care of deliberation and analysis, the extent to which the opinion formation process has been extensive and wide-ranging, free of censorship and independent of social pressures. At the collective level, the decision making process should ideally represent those affected, generate many different viewpoints and decision options, and be perceived as legitimate by all participants (see Price and Neijens Deliberative polls seek to maximize select quality criteria, as we will soon illustrate. But it is important to recognize that they are not the first attempts to do so. Indeed, deliberative polls historically follow a long line of efforts by pollsters and survey researchers to counter the tendency of mass opinion surveys to collect and disseminate opinions that may be ill-informed 'non-attitudes' or 'pseudo-opinions' (Price and Neijens 1997). Survey researchers and pollsters have long attempted to 'qualify' their measures of opinion in various ways. Examples include the use of 'don't know' response options, probes and question filters designed to reduce manufactured opinions, or the addition of measures of opinion intensity to accompany directional measures of opinion (e.g. Schuman and Presser 1981, Converse and Presser 1986). Application of 'likely voter' models to qualify poll results in election contexts (e.g. Traugott and Tucker 1984) or the use of a 'mushiness index' that attempts to assess the relative stability, consistency, and thoughtfulness of mass opinion data (Yankelovich 1991), are some other illustrations. Survey and polling methods aimed at increasing the quality of public opinion range from these sorts of attempts at filtering standard opinion data (some of which have become widely accepted as good research practice) to more concerted efforts to inform or educate survey respondents—what we will call surveys of informed public opinion. Even more ambitious are new techniques that aim to inform through lengthy programs of active deliberation—what we will call deliberative polls. Here we will examine both approaches, selecting in each case several of the most prominent and well-developed examples for review and 148 INTERNATIONAL JOURNAL OF PUBLIC OPINION RESEARCH analysis. Among surveys of informed opinion we will single out recent work by the Americans Talk Issues Foundation (ATIF, e.g. Kay et al. 1994b) and by Dutch researchers using the Choice Questionnaire (e.g. Neijens 1987). Among deliberative polls we will focus on the American and British Deliberative Polls organized by James Fishkin (e.g. Fishkin 1995) and the Planning Cell method developed in Germany by Peter Dienel (e.g. Dienel 1978, 1989). AMERICANS TALK ISSUES Since 1987 the Americans Talk Issues Foundation (ATIF) has conducted a series of'educational' public interest surveys on a variety of issues. The surveys are designed and conducted by small, politically balanced teams of issue experts and public opinion researchers (Kay et al. 1994a). The teams work to insure that both the questions and the analysis are fair and balanced and, within the limits of telephone interviewing, provide accurate contextual and factual information. The ATIF program of surveys has a number of quality objectives. One main goal is consensus location: 'A search for the most widely supported proposals by testing various features of the proposals separately and in combination, as well as verifying support by exposing respondents to pro and con arguments, including cost, benefits, and probable consequences of the most preferred proposals' (Kay et al. 1994a). More generally, the ATIF surveys aim at a variety of desired outcomes: achieving consensus (a collective outcome), increasing the extensiveness of the information base (an individual or collective outcome), and encouraging respondents to confront and accept the consequences of their opinions (an individual or collective outcome). They seek to optimize collective interests and better represent collective desires (collective outcomes). In that they are intended to assist policy makers, they also aim to improve the clarity of the path to action (a collective outcome). In terms of opinion-formation processes, the ATIF efforts aim at improving the extensiveness of the information search by political leaders as they formulate public policy proposals. The surveys are designed to extend mass participation in decision making by gathering public views of a broad range of policy options and potential solutions to social problems. Conventional opinion polls and surveys assess popular responses to a narrow range of proposals favored by elites, in the latter stages of decision making. In contrast, the ATIF program seeks ways of injecting mass input during the earlier, developmental phase when policy options are first being assembled (see Price and Neijens 1997). Some examples will illustrate how the surveys are conducted. In March 1993 and January 1994, ATIF examined public responses to a total of 50 proposals for government reform (ATIF Surveys 22 and 24). Three different approaches DELIBERATIVE POLLS 149 to gathering opinions were employed. First, some of the proposals were presented in pairs or triplets, randomly ordered, of slightly different versions (e.g. 'limiting terms of office of members of Congress and other elected officials,' versus 'limiting terms of elected officials, registered lobbyists, and government bureaucrats,' (Kay et al. 19944, p. 2). This was intended to allow analysts to detect the impact of small variations in wording, ideas, or the possibility of bias (p. 3). The second approach involved asking opinions on a baseline proposal (e.g. 'partial public funding to candidates running for Congress') followed by a series of limiting conditions (e.g. 'How would you rate this proposal . . . if you knew that: . . . Funding would come in part from a new check-off box on IRS tax returns'; Kay et al. 1994a, p. 5). The third approach consisted of asking respondents their opinion on a particular proposal (e.g. to 'require candidates for Congress to raise, at least, one-half of their campaign funds from individual voters in their districts') followed by a series of persuasive arguments supporting and opposing the proposal, most citing one or another desired or undesired consequence should the proposal be enacted. Respondents were asked to evaluate how convincing they found each argument to be. After these ratings, they were once again asked their opinion of the proposal (e.g. 'Now, that you have considered all these arguments, rate the original proposal again . . . Do you favor or oppose making this proposal into law?'; Kay et al. 1994a, p. 7). Thus, the ATIF formats are designed to assess what difference it makes for measured public support of particular proposals if survey respondents are fully aware of multiple options and encouraged to consider possible outcomes of each. T H E CHOICE QUESTIONNAIRE The Choice Questionnaire (Neijens 1987, Neijens et al. 1992) resembles in some respects the third approach employed by the ATIF program and shares many of its quality objectives. However, it eschews the argumentative format used by ATIF and employs a more detailed, face-to-face survey format. The Choice Questionnaire also limits its efforts to improving the evaluation phase of collective decision making rather than attempting to inject mass opinion into an earlier, policy-development stage. It attempts to provide citizens with a broad base of reasonably objective information, summarizing a full range of viable policy options and the probable consequences of each, provided by technical and policy experts. One of the Choice Questionnaire's distinguishing features is that it aims to insure, by design, that respondents fully understand the consequences of various options, think carefully about them, and make trade offs among consequences when deciding their choice. In doing so, the Choice Questionnaire provides a more highly structured information and response format than do the ATIF surveys. 150 INTERNATIONAL JOURNAL OF PUBLIC OPINION RESEARCH The instrument was developed for use in connection with the General Social Debate (GSD) in the Netherlands. The GSD, carried out from 1982 to 1984, focused on alternative national programs for electricity production. Initiated by the Dutch Government because of substantial extra-parliamentary opposition to the use of nuclear power, the GSD sought to involve the Dutch population in determining a national energy policy. A Steering Committee, consisting of widely respected members recruited from parties with different standpoints in the debate, was given the task of organizing the debate. The Steering Committee prepared an Interim Report containing information pertinent to the problems at hand, after consultations with interested organizations, popular action groups, and citizens attending organized public hearings and discussions. The Committee next attempted to gather the opinions of citizens from all strata of society. The Choice Questionnaire, which contained information based on the Interim Report, was one of the instruments used in this popular opinion-gathering process. Following the work of decision analysts (e.g. Keeney and Raiffa 1976, Edwards 1977, 1983), the Choice Questionnaire provided encapsulated information about principal energy policy options and their consequences, as described by a number of attributes—for example, costs, environmental impact, and social and health-related consequences. Respondents were presented with information cards describing six alternative energy strategies, five relying upon different sources of power generation (natural gas, oil, coal, nuclear power, and wind energy) plus a conservation strategy. A set of six to nine consequences was offered for each strategy. When experts disagreed over potential outcomes, their conflicting views were each represented. After reviewing the information, respondents had to select a combination of any three strategies, under a set of guidelines insuring that the three strategies together would provide sufficient energy. Twenty combinations were thus possible. Because of concerns about respondent 'information overload' (Slovic and Lichtenstein 1971), processing of the supporting material was aided by a serial evaluation procedure. Respondents were asked to evaluate the attractiveness of each consequence, with the expectation that they would absorb the information more thoroughly as a result of the series of judgments they had to make. Respondents were first asked whether they considered a given consequence to be important or unimportant. If they believed the consequence to be important, they were asked to indicate whether they considered it advantageous or disadvantageous, and to estimate the magnitude of the advantage or disadvantage. They were then asked to add up the advantageous and disadvantageous evaluations for each option, respectively (assuming that such a 'book-keeping system' would help them make their choice without having to recall all the information). They were, finally, asked to choose the preferred combination of DELIBERATIVE POLLS 151 three energy strategies. No prescription—that is, no decision rule—was given for this final choice. The procedures involved in application of the Choice Questionnaire, then, are more elaborate than those used in the ATIF telephone surveys. In a face-to-face general population survey, field interviewers gathered background information from respondents, presented the Choice Questionnaire, explained how to complete the evaluation packet, and assisted respondents in filling out a practice questionnaire. Interviewers then left respondents with a booklet containing the Choice Questionnaire, requested that it be filled out at home within a week, and later returned to collect it. T H E DELIBERATIVE POLL More elaborate than either of the two surveys of informed opinion so far described is the Deliberative Poll (Fishkin 1991, 1995). According to Fishkin, 'an ordinary poll models what the electorate thinks, given how little it knows. A deliberative opinion poll models what the electorate would think if, hypothetically, it could be immersed in intense deliberative processes' (Fishkin, 1991, p. 81; italics in original). The basic plan is to select a national, random sample of the voting-age population and transporting them as 'delegates' to a single site for several days of debate and deliberation. The delegates debate issues with political leaders and each other, and are then polled on their preferences. The general design is modeled on the ancient, Athenian practice of selecting, by lottery, citizen juries numbering around 500 to judge accusations or charges against political leaders. This sort of 'court of public opinion' enjoyed great power in Athens, even in certain cases the right to review and overturn decisions of the Athenian Assembly (Fishkin 1991, pp. 87-8).' The process is intended 'to adapt the deliberative possibilities of small-scale politics to the problem of selecting candidates and launching issues in a large-scale nation state' (Fishkin 1991, p. 8). The exercise would combine the thoughtfulness and depth f face-to-face politics with the representative character of a national event (p. c, 1. The Deliberative Poll focuses on most of the same quality objectives as the ATIF surveys and the Choice Questionnaire, but adds to these the goals of advancing discussion and debate as both individual and collective processes, and to create a communicative forum that is fully representative of the population at large. The Deliberative Poll aims at improving the extensiveness of the information search among political leaders and the public alike. Like the ATIF surveys, it 1 It is perhaps worth noting, (as docs Fishkin 1991), that it was just such a randomly constituted citizen jury that condemned Socrates to death, a fact that underscores the complicated relationship between the quality of a decision making process and the quality of decision outcomes. 152 INTERNATIONAL JOURNAL OF PUBLIC OPINION RESEARCH seeks to increase mass participation in the proposal development process, by creating an event that would allow ordinary citizens to penetrate the 'hocus pocus' and 'amplification' of press-mediated political debate and engage directly with their leaders (Fishkin 1991, p. 8). This discussion, argues Fishkin (1991, p. 92) cannot merely be one-way. 'The basic point,' says Fishkin, is that deliberative opinion polls offer direct democracy among a group of politically equal participants who, as a statistical microcosm of the society, represent or stand in for the deliberations of the whole. The institution is, in that sense, a direct face-to-face society for its participants and a representative institution for the nation-state (1991,